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Abstract  Segmenting  materials’  images  is  a  laborious  and 
time-consuming  process,  and  automatic  image  segmentation 
algorithms  usually  contain  imperfections  and  errors.  Inter¬ 
active  segmentation  is  a  growing  topic  in  the  areas  of  image 
processing  and  computer  vision,  which  seeks  to  find  a  balance 
between  fully  automatic  methods  and  fully-manual  segmen¬ 
tation  processes.  By  allowing  minimal  and  simplistic  inter¬ 
action  from  the  user  in  an  otherwise  automatic  algorithm, 
interactive  segmentation  is  able  to  simultaneously  reduce  the 
time  taken  to  segment  an  image  while  achieving  better  seg¬ 
mentation  results.  Given  the  specialized  structure  of  materi¬ 
als’  images  and  level  of  segmentation  quality  required,  we 
show  an  interactive  segmentation  framework  for  materials’ 
images  that  has  three  key  contributions:  (1)  a  multi-labeling 
approach  that  can  handle  a  large  number  of  structures  while 
still  quickly  and  conveniently  allowing  manual  addition  and 
removal  of  segments  in  real-time,  (2)  multiple  extensions  to 
the  interactive  tools  which  increase  the  simplicity  of  the  inter¬ 
action,  and  (3)  a  web  interface  for  using  the  interactive  tools 
in  a  client/server  architecture.  We  show  a  full  formulation  of 
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each  of  these  contributions  and  example  results  from  their 
application. 
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1  Introduction 

Interactive  segmentation  is  a  rapidly  growing  area  of  com¬ 
puter  vision  and  has  seen  heightened  interest  recently 
[28,49].  While  traditional  segmentation  seeks  to  identify 
objects/structures  within  an  image  in  a  fully  automated  fash¬ 
ion,  interactive  segmentation,  similar  to  active  learning  [43], 
accomplishes  the  goal  of  image  segmentation  while  incorpo¬ 
rating  a  sparse  number  of  user  interactions  which  are  included 
as  additional  constraints  or  guidance  in  the  segmentation 
model  or  algorithm.  These  interactions  may  take  on  differ¬ 
ent  forms,  and  may  include  drawing  a  bounding  box  [39], 
roughly  outlining  a  boundary  [32],  or  drawing  brush  strokes 
inside  and/or  outside  the  object  of  interest  [3,41,54,57].  A 
desired  property  of  an  interactive  segmentation  approach  is 
that  the  user  interaction  be  as  convenient  (i.e.,  low  cogni¬ 
tive  load)  and  sparse  (i.e.,  few  in  number)  as  possible,  while 
simultaneously  providing  immediate  feedback  to  the  user  on 
every  interaction. 

One  domain  that  has  been  unaddressed  in  interactive  seg¬ 
mentation  literature  is  materials  science  image  segmentation, 
where  there  are  no  existing  techniques  focusing  solely  on 
segmenting  materials’  images  using  an  interactive  approach. 
Materials  science  is  especially  important  to  the  development 
of  new  metals  and  biomaterials,  and  presents  unique  chal¬ 
lenges  in  image  segmentation.  First,  materials’  images  often 
are  3D  volumes  [21]  made  up  of  a  sequence  of  individ¬ 
ual  2D  image  “slices,”  as  shown  by  the  two  sample  slices 
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Fig.  1  Two  adjacent  slices  of  a  titanium  image  volume  [40].  Image  intensity  inverted  for  clarity;  the  original  is  similar  in  appearance  to  Fig.  1 1 


in  Fig.  1  1 .  This  large  number  of  slices  must  all  be  seg¬ 
mented  to  fully  and  properly  analyze  the  3D  structure  of 
the  material.  Second,  depending  on  the  inter-slice  distance, 
the  2D  structure  in  two  neighboring  slices  may  show  high 
continuity.  Such  inter- slice  structure  continuity  must  be  con¬ 
sidered  to  achieve  accurate  segmentation.  Third,  materials 
volumes  consist  of  numerous  substructures  (e.g., “grains” 
in  a  metallic  material,  or  “cells”  in  a  biomaterial,  etc.) 
with  complex  relationships  (e.g.,  adjacency/nonadjacency 
relationships)  among  them  that  determine  many  desir¬ 
able  properties  of  the  material  [38,51].  Existing  interactive 
segmentation  techniques  often  only  focus  on  foreground- 
background  segmentation  [3,39],  and  may  not  scale  to  the 
large  number  of  substructures  present  in  materials’  images. 
Other  methods  may  handle  multiple  structures  [49,50],  but 
do  not  incorporate  any  prior  knowledge  about  the  unique  rela¬ 
tionships  among  substructures  in  materials’  images  [37,52]. 
Finally,  the  imaging  techniques  used  to  obtain  a  materials 
image  volume  may  result  in  significant  noise  or  other  ambi¬ 
guities  that  increases  the  difficulty  to  segment  a  materials 
image  volume  in  a  fully  automatic  fashion.  Many  of  these 
properties  are  not  unique  to  materials’  images,  e.g.  medical 
images  may  exhibit  complex  relationships,  neuroimages  may 
have  high  continuity  between  slices,  natural  images  may  con¬ 
tain  numerous  structures,  etc.  and  as  such,  advances  in  inter¬ 
active  segmentation  may  have  far-reaching  consequences. 

In  this  paper,  we  present  an  interactive  segmentation 
approach  to  segment  materials  science  image  volumes.  We 
show  that  an  existing  propagation-based  materials  image  seg¬ 
mentation  approach  [58]  can  be  extended  to  allow  for  con¬ 
venient  interactive  segmentation.  We  illustrate  the  perfor¬ 
mance  of  the  proposed  approach  by  using  it  to  segment  a 
materials  image  volume  using  smaller  number  of  interac¬ 


1  For  clarity,  we  inverted  the  image  intensity  in  this  figure,  as  well 
as  several  other  figures  in  the  later  sections.  The  original  is  similar  in 
appearance  to  Fig.  11. 


tions  compared  with  general-purpose  interactive  segmenta¬ 
tion  methods  that  do  not  incorporate  materials- specific  priors. 
Further,  we  develop  extensions  to  the  interactive  tools  we 
present-namely,  annotation  repropagation,  parameter  esti¬ 
mation,  and  salient  region  detection  extensions-and  show 
that  these  extensions  lead  to  improved  performance.  Finally, 
we  illustrate  the  client/server  architecture  used  to  build  a 
web  application  for  implementing  the  proposed  approach  and 
related  extensions. 


1 . 1  Related  work 

Though  no  existing  work  focuses  on  the  materials  image  seg¬ 
mentation  application  using  interactive  segmentation,  we  dis¬ 
cuss  related  work  that  focuses  on  either  of  these  two  aspects. 

There  are  a  number  of  existing,  non-interactive  approaches 
to  segment  materials’  images  [10,48].  Among  the  most 
prominent  is  the  work  of  Comer  et  al.  [12,13]  on  the 
EM/MPM  algorithm,  originating  from  [30].  Other  meth¬ 
ods  that  have  been  specifically  used  on  materials’  images 
include  graph  cut  [20,58],  stabilized  inverse  diffusion  equa¬ 
tions  [19],  Bayesian  methods  [1 1,47],  and  the  watershed  [29] 
method.  Most  often,  materials’  images  are  opportunistically 
segmented  by  the  simplest  tools  available,  such  as  thresh¬ 
olding  [17,45],  or  out-of-the-box  methods  such  as  water¬ 
shed  or  normalized  cut  [46].  However,  these  methods  do  not 
incorporate  any  interaction  for  manual  refinement  by  a  user. 
Some  of  these  approaches  may  require  significant  time  to 
run;  requiring  the  user  to  examine  and  correct  errors  only 
after  the  algorithm  is  complete  may  not  be  practical  if  rapid 
refinement  is  desired.  Conversely,  the  general-purpose  inter¬ 
active  segmentation  techniques  discussed  previously  do  not 
incorporate  any  specific  domain  knowledge  about  materials’ 
images,  and  thus  may  require  additional  effort  on  the  part  of 
the  user  than  may  otherwise  be  needed  when  segmenting  a 
materials’  image  volume. 
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Interactive  grain  image  segmentation 


General  interactive  methods  exist  to  segment  the  object  of 
interest  using  a  model  learned  from  user  interactions  [3,39, 
54].  Other  approaches  incorporate  interaction  into  morpho¬ 
logical  operations  (watershed)  [49],  co- segmentation  [1],  or 
incorporate  machine-learning  to  aid  in  the  interactive  process 
[28,53].  These  interactive  methods  have  been  applied  to  a 
number  of  domains,  including  natural  images  [39],  medical 
images  [4],  and  neuroimages  [49,50]. 

Interactive  methods  that  focus  on  segmenting  3D  volumes 
share  elements  in  common  with  the  proposed  work.  The 
majority  of  these  related  works  are  in  the  medical  imaging 
area.  [18]  minimizes  an  energy  function,  though  the  formu¬ 
lation  is  different  from  the  proposed  approach.  Both  [9,53] 
employ  a  GPU  implementation,  however  [9]  includes  a  learn¬ 
ing  component,  as  we  do  in  Sect.  7.  Most  similar  to  the  pro¬ 
posed  method,  [33]  uses  a  propagation  framework  to  do  3D 
interactive  segmentation  on  medical  images.  A  graph  cut- 
based  minimization  is  used  by  [2],  as  is  used  in  this  paper. 
Finally,  [22]  introduce  a  web  interface  for  interactive  seg¬ 
mentation,  as  we  do  in  this  paper.  All  of  [2,9, 18,22,33,53] 
are  specific  to  medical  applications,  and  thus  are  targeted  at 
a  single  or  small  number  of  segments,  and  are  not  able  to 
handle  large  numbers  of  segments  with  different  adjacency 
relations,  as  we  do  in  this  paper. 

There  are  also  a  number  of  general  3D  interactive  meth¬ 
ods  that  are  not  specific  to  medical  images.  Among  these 
is  [26],  which  is  designed  specifically  to  divide  an  existing 
3D  mesh,  and  does  not  apply  to  the  serial-sectioned  3D  vol¬ 
umes  used  in  this  paper.  Further,  [25]  builds  a  tool-focused 
framework  that  includes  “hole  filling,”  “point-bridging,”  and 
“surface-dragging”  tools.  We  similarly  develop  a  tool-based 
approach  along  with  multiple  extensions,  but  we  use  a  dif¬ 
ferent  approach  (addition  and  removal)  for  the  interaction  in 
this  paper.  Finally,  most  similar  to  the  materials  application  in 
this  paper,  the  proprietary  INTERSEG  [35]  plugin  for  Image J 
[42]  focuses  on  interactively  segmenting  cell-like  structures. 
These  structures,  however,  have  different  types  of  adjacency 
relations  compared  with  the  grain-structures  discussed  in  this 
paper,  and  thus  the  INTERSEG  plugin  uses  different  tools  for 
interaction  that  we  introduce  in  Sects.  3  and  4. 

The  remainder  of  this  paper  is  organized  as  follows:  In 
Sect.  2  we  discuss  the  background  for  the  proposed  inter¬ 
active  segmentation  approach  for  materials  image  volumes. 
We  further  discuss  how  the  proposed  approach  handles  seg¬ 
ment  removal  in  Sect.  3  and  segment  addition  in  Sect.  4.  The 
first  extension  of  the  proposed  method  in  Sect.  5  discusses 
how  adjacent  slice  similarity  can  be  leveraged  to  allow  for 
segment  repropagation  to  reduce  the  number  of  interactions 
required.  In  Sect.  6,  we  show  how  some  of  the  parameters 
of  the  proposed  method  can  be  automatically  estimated.  The 
final  extension  of  the  proposed  approach  in  Sect.  7  shows  how 
a  simple  online  learning  system  can  detect  salient  regions  that 
should  be  given  extra  attention  by  a  human  annotator.  In  Sect. 


9,  we  evaluate  the  proposed  method’s  performance  against 
another  general-purpose  interactive  segmentation  method. 
Finally,  in  Sect.  10  we  provide  brief  concluding  remarks. 


2  Interactive  materials  image  segmentation 


In  [58]  we  developed  a  3D  materials  science  image  segmen¬ 
tation  method  by  propagating  segmentation  S u  of  a  slice  U 
to  a  neighboring  slice  V,  resulting  in  a  segmentation  Sv . 
This  way,  using  an  initial  segmentation  on  one  slice,  we 
can  repeatedly  propagate  this  segmentation  to  the  remaining 
slices  in  the  volume  to  obtain  a  complete  3D  segmentation. 
This  propagation  was  done  while  preserving  the  topology 
(i.e.,  non-adjacency  relations  among  2D  segments),  which 
led  to  a  better  performance  when  compared  with  methods 
that  did  not  incorporate  topology  as  a  prior.  Specifically,  let 
the  segmentation 

Su  =  {SY,S%9...,S%}, 

where  ,  i  =  \...n  are  disjoint  segments  in  slice  U, 
and  this  collection  of  segments  makes  up  a  partition  of  the 
slice  U, 

u  =  \Js?. 

i= 1 

An  example  is  shown  in  Fig.  2  where  all  the  segments  (“grain” 
structures)  are  separated  by  red  lines.  To  propagate  segmen¬ 
tation  S u  to  a  new  slice  V  to  yield  the  segmentation  Sv ,  we 
minimize  the  energy 

E(Sv)=J^&p(S^)+  X  ^pgiS^Sj)  (1) 

P*v  {p,q}eVX 


where  is  the  set  of  all  four-connected  pixels  in  V.  The 
unary  term  ®p(S^ ),  which  represents  a  cost  for  a  pixel  p 
being  assigned  to  a  segment  Sj  in  slice  V,  was  set  to  reflect 
the  structure  continuity  between  U  and  V , 


®p(S?) 


0,  distance (p,  S^)  <  d 
oo,  otherwise 


(2) 


where  d  is  a  dilation  distance  that  reflects  the  maximum  pos¬ 
sible  structural  change  between  U  and  V  [58] .  In  addition,  the 
binary  term  <&pq  (S^ ,  Sj),  which  represents  a  cost  for  a  pair 
of  neighboring  pixels  p,  q  being  assigned  to  two  (possibly 
the  same)  segments  Sj ,  S  J ,  was  constrained  to  preserve  non- 
adjacency  segment  relationships  from  U  to  V;  i.e.,  any  two 
segments  ,  Sj  are  allowed  to  be  adjacent  (have  pixels  that 
are  four-connected  between  them)  only  if  the  corresponding 
segments  Sj7 ,  S1/  are  also  adjacent, 

1  j 
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Fig.  2  Example  of  segmentation  propagation,  highlighting  different  appear  in  V  {yellow),  an  erroneously  missing  segment  {blue)  and  a 

types  of  topology  changes,  including  segments  that  become  unadjacent  spurious  segment  {green).  Image  intensity  inverted  for  clarity 

in  V  {S2,  S3),  segments  that  disappear  in  V  {purple),  segments  that 


<$>Pq(Sj,S]) 


0,  i  =  j 

00,  {S’J,  sV }  i  Au  ,  (3) 

g(p,q),  {S’J,  Sf\  e  Au 


where  Au  contains  segment  pairs  that  are  adjacent  in  Su ,  and 
we  set  g{p,  q)  to  reflect  the  image  boundary  information  in 

V  [58].  An  example  is  shown  in  Fig.  2,  where  and 
are  allowed  to  be  adjacent  because  Sf7  and  S^7  are  adjacent 
in  Su .  However,  and  Sj  are  not  allowed  to  be  adjacent 
(have  an  infinity  penalty)  because  Sf7  and  are  not  adjacent 
in  Su .  This  topology  constraint  was  found  to  be  particularly 
important  for  materials’  images,  and  our  proposed  method 
was  able  to  outperform  other  methods  that  did  not  incorporate 
such  a  prior. 

While  finding  the  global  minimum  to  this  cost  is  NP-hard, 
this  cost  has  been  shown  to  be  minimizable  to  a  local  opti¬ 
mum  using  a  graph-cut  approach  [6,55].  However,  one  phe¬ 
nomenon  that  was  observed  in  this  previous  work  was  that, 
during  propagation,  2D  structure  topology  between  U  and 

V  might  not  always  be  fully  consistent.  For  example,  a  new 
3D  structure  with  no  intersection  in  slice  U  might  appear  in 
slice  V ,  e.g.,  the  structure  in  the  yellow  circle  in  Fig.  2.  Simi¬ 
larly,  a  3D  structure  intersected  by  slice  U  might  disappear  in 
slice  V ,  such  as  the  structure  circled  in  magenta  in  Fig.  2.  This 
breaks  the  topology  constraints  given  in  Eq.  (3)  in  some  local 
regions.  This  may  lead  to  spurious  segments  and  missing 
structures,  as  circled  in  green  and  blue  respectively,  in  Fig.  2. 

Our  method  made  use  of  a  brute-force  automated  search  to 
locate  such  spurious  and  missing  structures  in  V  [58] .  Given  a 
large  number  of  substructures  in  a  material  sample,  it  is  com¬ 
putationally  expensive  to  examine  every  location  for  possible 
spurious  or  missing  structures.  Furthermore,  given  the  small 


inter-slice  distance,  spurious  or  missing  structures  are  usu¬ 
ally  very  small  and  thus  difficult  for  an  automatic  algorithm  to 
segment  correctly.  In  this  paper,  our  goal  is  to  develop  effec¬ 
tive  interactive  tools  to  allow  a  user  to  conveniently  specify 
the  local  areas  that  contain  spurious  or  missing  structures,  and 
incorporate  such  interactions  to  refine  the  segmentation  Sv  to 
a  corrected  Sv  on  slice  V ,  using  the  same  energy  minimiza¬ 
tion  algorithm.  More  specifically,  we  propose  to  allow  the 
user  to  correct  these  two  types  of  segmentation  errors  within 
this  segmentation  propagation  framework  by:  (a)  annotating 
the  location  of  a  new  segment  to  handle  cases  where  a  new 
structure  appears  in  slice  V ,  and  (b)  annotation  of  existing 
segments  that  should  no  longer  be  present  in  segmentation 

These  interactions  are  inherently  local  because  the  2D 
cross  section  of  a  3D  structure  shows  very  small  size  before 
appearing  or  disappearing  from  a  neighboring  2D  slice. 
Therefore,  correcting  Sv  to  Sv  can  be  achieved  by  using  the 
same  energy  minimization  in  local  image  areas  around  the 
interactive  annotations.  This  is  also  important  because  inter¬ 
active  segmentation  requires  instantaneous  user  feedback. 
The  previous  propagation  method  segmented  entire  slices, 
which  was  more  computationally  intensive  than  is  desirable 
in  an  interactive  system.  We  will  further  discuss  these  two 
interactions,  and  how  we  identify  local  regions  for  each,  in 
the  following  subsections. 


3  Removal  of  spurious  segments 

For  this  interaction,  we  allow  the  user  to  select  a  spurious 
segment  for  removal  by  clicking  the  mouse  on  this  seg¬ 
ment  in  a  visualized  segmentation  of  Sv .  Instead  of  naively 
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(a)  (b)  (c) 

Fig.  3  Example  selection  of  a  spurious  segment  for  removal,  a 
Chosen  and  surrounding  segments,  b  Local  region  extracted  around 
.  c  The  updated  segmentation  in  the  extracted  local  region 


removing  this  segment  by  arbitrarily  merging  it  into  one  of  its 
neighbors,  we  use  the  same  energy  minimization  discussed 
above  to  assign  the  individual  pixels  in  to  potentially  dif¬ 
ferent  neighboring  segments.  As  discussed  above,  we  iden¬ 
tify  a  local  region  in  which  we  update  the  segmentation. 
Specifically,  this  local  region  consists  of  the  specified 
and  its  neighboring  segments,  e.g.,  ,  S \ surrounding 

the  spurious  segment  in  Fig.  3a,  and  re-run  the  energy 
minimization  within  this  local  region  after  modifying  the  © 
term  to  incorporate  the  interaction,  resulting  in  an  updated 
segmentation  in  this  local  region,  as  shown  by  the  example 
in  Fig.  3c.  For  ease  of  notation,  we  use  similar  notation  to 
the  adjacency  definition  in  Eq.  (3)  by  using  {Av  }k  to  refer  to 
the  set  of  segments  neighboring  the  segment  .  This  way, 
the  local  region  for  updating  the  segmentation  is 


C  =  {Avh\JsV.  (4) 

In  this  local  region,  we  rerun  the  energy  minimization 
of  Eq.  (1)  by  modifying  the  ©  term.  In  particular,  we  do  not 
allow  any  pixel  to  be  assigned  to  S %  since  this  segment  is  to  be 
removed.  Instead,  the  pixels  initially  in  S %  can  be  assigned  to 
any  of  the  segments  in  {Av  }k  with  0  cost  for  the  ©  term,  i.e., 


Vp  e  y.  ®p(Sj) 


0,  Sj  e  {Avh 

oo,  otherwise 


Vp£Svk,  ®p{Sj)  =  @p{Sj) 


(5) 


By  updating  ©  in  this  fashion,  we  do  not  require  the  pixels 
previously  in  S %  merged  into  a  single  neighboring  segment. 
Instead,  these  pixels  may  be  assigned  to  more  than  one  seg¬ 
ment  in  { Av)k ,  as  shown  in  Fig.  3c. 

Note  that  this  interaction  is  very  simple  and  convenient, 
as  it  requires  only  a  single  click  anywhere  inside  the  spuri¬ 
ous  segment  S % .  The  full  algorithm  for  removing  spurious 
segments  is  summarized  in  Algorithm  1 . 


4  Addition  of  missing  segments 

Unlike  removal,  interactively  annotating  an  additional  struc¬ 
ture  cannot  be  solely  formulated  as  a  simple  modification 


Algorithm  1  Interactively  specifying  segment  to  remove. 

1:  function  RemoveSegment^  ,  S l ) 

2:  C  <-  {AV}k  U  SVk 

3:  For  the  pixels  C,  build  graph  for  energy  minimization 

problem  from  Section  2 
4:  ©  set  from  Eq.  (5) 

5:  Sv  <r-  Sv  incorporating  the  updated  segmentation  in 

C 

6:  return  updated  Sv 

7:  end  function 


(a)  (b)  (c) 


Fig.  4  Annotating  the  addition  of  a  missing  segment,  a  Segmentation 
Sv  with  a  missing  segment  near  the  center  of  the  image,  b  Annotation 
of  a  center  point  c,  along  with  a  seed  radius  s  and  a  dilation  radius  d,  and 
the  identified  local  region  for  updating  the  segmentation,  c  The  updated 
segmentation  of  the  local  region  shown  in  (b) 


of  the  ©  term  in  the  energy  minimization  formulation.  This 
is  because  the  multi-labeling  problem  used  to  optimize  the 
energy  minimization  form  in  Eq.  (1)  optimizes  over  a  fixed 
set  of  segments,  and  cannot  introduce  new  segments.  Thus, 
for  each  missing  segment,  we  must  explicitly  create  a  new 
segment  at  the  location  interactively  specified  by  the  user. 

Based  on  the  initial  segmentation  Sv  =  {S±  ,  Sj, 

. . . ,  },  we  take  as  input  from  the  user  an  annotation  speci¬ 

fying  the  center  location  c  of  the  new  segment  S^+l .  In  addi¬ 
tion  to  this,  we  also  accept  two  parameters  from  the  user:  (1) 
the  seed  radius  s  specifying  a  circular  region  around  c  such 
that  this  circular  region  is  completely  contained  within  the 
missing  structure;  (2)  a  dilation  radius  d ,  which  is  similar  to 
the  dilation  parameter  used  in  Sect.  2,  such  that  the  circular 
region  with  this  dilation  radius  d  centered  at  c  completely 
covers  the  missing  structure  to  be  segmented.  We  explicitly 
enforce  that  d  >  s  for  any  choice  of  s .  We  call  pixels  within 
the  seed  radius  s  of  c  “seed  pixels”  and  pixels  within  the 
dilation  radius  d  of  c  “dilation  pixels.”  In  this  interaction, 
seed  pixels  are  guaranteed  to  be  part  of  the  missing  segment 
that  is  added,  as  shown  by  the  green  circle  in  Fig.  4b,  and 
dilation  pixels,  excluding  seed  pixels,  are  potentially  part  of 
the  missing  segment  that  is  added,  as  shown  by  the  blue  area 
in  Fig.  4b.  This  makes  the  selection  of  s  and  d  conceptually 
simple  for  the  user  to  tune.  In  Sect.  6,  we  discuss  how  to 
automate  the  selection  of  s  and  d  to  further  reduce  the  user’s 
burden  when  interactively  segmenting  a  materials  volume. 
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Algorithm  2  Interactively  specifying  segment  to  add. 

1:  function  ADDSEGMENT(Sy ,  c ,  s,  d) 

2:  C  union  of  all  segments  that  contain  a  seed  pixel  or 

dilation  pixel 

3 :  For  the  pixels  in  C ,  build  graph  for  energy  minimization 

problem  from  Section  2 
4:  ©  <r-  set  from  Eq.  (6)  and  Eq.  (7) 

5:  Sv  <r-  Sv  incorporating  the  updated  segmentation  in 

C 

6:  return  updated  Sv 

7:  end  function 


Fig.  5  Alternate  annotation  shapes  for  addition,  a  Standard  point- 
annotation.  b  Line  annotation,  c  “Scribble”  annotation.  Dark  areas  are 
the  seed  pixels  and  grey  areas  are  the  dilation  pixels 


Similar  to  the  removal  interaction  in  Sect.  3,  we  define 
a  local  region  around  the  specified  c  to  update  the  segmen¬ 
tation  of  Sv .  Specifically,  we  define  this  region  by  taking 
all  segments  in  Sv  that  contain  one  or  more  seed  or  dila¬ 
tion  pixels.  In  this  local  region  we  modify  the  ©  term  of  the 
energy  minimization  in  Eq.  (1)  to  obtain  an  updated  segmen¬ 
tation.  Specifically,  we  allow  all  seed  and  dilation  pixels  to 
be  reassigned  to  the  new  segment  Sj+1  by  setting 


©P(5„y+i)  = 


0,  \\P-C\\<d 

oo,  otherwise 


(6) 


where  ||  •  ||  is  the  Euclidean  distance  between  pixels  p  and  c. 
Furthermore,  to  ensure  that  the  seed  pixels  are  always  guar¬ 
anteed  to  be  part  of  S^+l  we  set  an  infinity  penalty  for  seed 
pixels  assigned  to  any  segment  other  than 

oo,  \\p  —  c\\  <  s  and  i  ^  n  +  1 

(7) 

(. sy ) ,  otherwise. 

The  full  algorithm  for  adding  a  missing  segment  is  summa¬ 
rized  in  Algorithm  2. 

Note  that  annotations  need  not  be  constrained  to  a  sin¬ 
gle  point,  as  shown  in  Fig.  5a.  Line-based  annotations,  as 
shown  in  Fig.  5b  can  be  defined  by  setting  seed  pixels  to 
be  all  those  pixels  within  a  distance  s  of  any  position  along 
an  annotated  line.  Dilation  pixels  can  be  defined  similarly. 
Further,  any  “scribble”-like  annotation,  provided  it  remains 
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connected  and  without  holes,  can  be  morphologically  dilated 
by  s  or  d  to  obtain  seed  or  dilation  pixels,  respectively,  as 
shown  in  Fig.  5c. 


5  Annotation  repropagation 

While  annotations  defining  addition  or  removal  of  segments 
can  be  made  on  a  single  slice,  they  may  provide  valuable 
information  for  adjacent  slices  and,  as  such,  can  be  propa¬ 
gated  (which  we  refer  to  as  “repropagation”)  to  these  adjacent 
slices,  similar  to  the  segmentation  propagation  discussed  in 
Sect.  2.  For  removal,  we  simply  locate  the  same  segment  in 
all  adjacent  slices  (if  present),  and  repeat  the  removal  opera¬ 
tion  summarized  in  Algorithm  1 .  For  addition,  illustrated  in 
Fig.  6,  we  create  the  new  segment  as  summarized  in  Algo¬ 
rithm  2,  after  which  we  repropagate  the  segmentation,  includ¬ 
ing  the  newly  created  segment,  to  the  adjacent  slices  in  the 
same  manner  as  discussed  in  Sect.  2.  This  repropagation  can 
be  done  within  the  local  region  C  on  the  adjacent  slices  for 
efficiency,  and  terminates  if  the  new  segment  is  no  longer 
present  during  the  repropagation  (determined  by  the  energy 
minimization  discussed  in  Sect.  2). 

When  errors  are  small,  such  annotation  repropagation  may 
not  be  particularly  beneficial.  However,  when  there  are  gross 
errors,  annotation  repropagation  can  reduce  the  number  of 
needed  annotations  significantly. 


ir1  u1  ui+1 

Fig.  6  Annotation  repropagation  for  addition.  The  blue  annotation  on 
slice  Ul  (row  1)  yields  an  updated  segmentation  of  slice  Ul  (row  2).  This 
new  segmentation  can  be  repropagated  to  the  neighboring  slices  Ul~l 
and  Ul+l  (row  3)  using  the  same  propagation  approach  introduced  in 
Sect.  2.  Image  intensity  inverted  for  clarity 
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6  Parameter  estimation 

When  interactively  adding  a  new  segment,  as  discussed  in 
Sect.  4,  the  seed  radius  s  and  dilation  radius  d  are  required 
to  be  specified  by  the  user.  This  results  in  additional  burden 
on  the  part  of  the  user.  In  this  section,  we  develop  a  para¬ 
meter  estimation  approach  to  automatically  select  these  two 
parameters  so  the  user  need  only  override  them  in  very  rare 
cases,  or  not  at  all. 

We  do  this  by  leveraging  information  about  the  center  c 
the  user  provided  relative  to  the  initial  segment  in  which  it 
resides.  Generally  a  missing  segment  occurs  when  2D  cross- 
section  intersects  a  new  3D  structure  in  V.  Given  a  small 
inter-slice  distance,  as  discussed  in  Sect.  2  where  neighbor¬ 
ing  slices  have  large  similarity,  we  expect  that  these  miss¬ 
ing  segments  are  often  small  compared  with  its  neighbor¬ 
ing  segments  in  slice  V.  An  example  is  shown  in  Fig.  7a, 
where  a  small  segment  is  missing  (indicated  by  the  yellow 
circle)  in  the  segmentation  Sv  \  this  missing  segment  is  mis¬ 
takenly  merged  into  a  large  neighboring  segment  .  Intu¬ 
itively,  placing  c  near  the  boundary  of  likely  indicates  the 
missing  segment  is  small,  as  shown  by  Fig.  7b.  Conversely, 
placing  c  closer  to  the  center  of  likely  indicates  the  result¬ 
ing  missing  segment  is  large  as  shown  in  Fig.  7c.  We  make 


(a)  (b) 


(c)  (d) 

Fig.  7  Automatic  selection  of  seed  radius  s  and  dilation  radius  d.  a 
A  missing  segment  located  within  a  large  segment  .  b,  c  Selections 
of  c  at  varying  distances  from  the  boundary  of  S % ,  resulting  in  different 
estimations  of  5.  d  Updated  Sv  by  adding  a  missing  segment  using  the 
seed  center  c  shown  (b)  and  the  proposed  parameter  estimation  method 
to  determine  s.  Image  intensity  inverted  for  clarity 


a  simplifying  assumption  that  we  do  not  allow  the  missing 
segment  to  spill  over  the  boundary  of  S % .  For  example,  the 
selection  of  c  and  s  in  Fig.  7b  is  able  to  generate  the  updated 
segmentation  shown  in  Fig.  7d. 

To  obtain  an  estimation  of  s  we  start  by  setting  s  =  0, 
and  we  then  increase  s  by  a  small  €  amount  until  the  cir¬ 
cle  centered  at  c  with  radius  s  is  within  e  distance  of  the 
boundary  of  the  containing  segment  ,  as  shown  by  the 
arrow  in  Fig.  7b,  c.  In  materials’  images,  the  majority  of 
newly  appearing  structures  when  moving  from  one  slice  to 
another  are  usually  near  the  boundary  of  an  existing  structure 
Sfr  (near  a  “Y”-junction  boundary  between  structures).  This 
automatic  approach  is  ideally  suited  for  these  cases.  When 
the  user  specifies  a  c  that  falls  directly  on  a  segment  bound¬ 
ary  in  Sv ,  we  default  to  requiring  user-supplied  s  in  these 
less-common  cases.  For  estimating  J,  it  is  scaled  according 
to  the  value  of  s.  Specifically,  we  set  d  =  2  -  s.  As  shown  in 
Sect.  9,  this  approach  saves  both  time  and  effort. 

7  Salient  region  detection 

Because  materials’  images  can  be  very  large  and  complex,  it 
can  take  a  significant  amount  of  time  for  a  human  annotator 
to  review  the  segmentation  of  such  a  large  image  to  determine 
where  it  may  require  additional  interaction.  In  this  section,  we 
introduce  a  salient  region  detection  approach  that  identifies 
candidate  regions  highlighting  the  areas  most  likely  to  need 
additional  interaction.  In  this  work,  we  define  a  salient  region 
as  a  subset  of  the  complete  image  that  is  most  likely  to  require 
interactivity,  i.e.  where  the  segmentation  shows  large  uncer¬ 
tainty.  Since  the  addition  of  missing  segments  is  the  most 
time-consuming  annotation  task,  we  focus  on  detecting  the 
edges  in  the  image  that  are  not  identified  as  segment  bound¬ 
aries,  indicating  a  missing  segment,  for  this  salient  region 
detection.  As  such,  we  identify  prominent  edges  fragments 
that  are  not  segmented  during  the  propagation  as  candidate 
regions,  and  use  a  SVM  classifier  [14]  to  learn  which  candi¬ 
dates  are  truly  salient  regions,  and  which  are  noise  that  can 
be  ignored  by  the  human  annotator.  These  salient  regions  are 
later  enclosed  in  a  bounding  box  for  easier  visualization. 

More  specifically,  we  use  the  online  learning  [44]  system 
outlined  in  Fig.  8.  We  first  use  the  Canny  edge  detector  [8]  in 
areas  that  are  a  fixed  distance  from  the  segmentation  bound¬ 
aries,  preventing  edges  in  the  image  which  already  fall  on 
segmentation  boundaries  from  being  considered  candidates, 
effectively  leaving  “residual”  edges,  which  may  correspond 
to  either  edges  of  missed  structures  or  noise.  The  output  of 
the  edge  detector  is  dilated  slightly  and  then  separated  into 
connected  components  to  produce  candidate  salient  regions, 
which  are  then  classified  using  an  initial  pool  of  300  posi¬ 
tive  and  negative  samples  to  determine  which  should  be  pre¬ 
sented  as  salient  regions.  Finally,  when  the  user  is  presented 
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Fig.  8  Salient  region  detection  pipeline.  Edges  are  detected  and  clas¬ 
sified  as  salient  or  non-salient.  Those  classified  as  salient  are  then  shown 
in  a  bounding  box  displayed  on  the  image  to  guide  the  interactive  seg- 


with  the  salient  regions  highlighted  on  the  segmentation,  any 
subsequent  interactive  annotations  are  converted  to  positive 
samples,  and  remaining  unannotated  salient  regions  are  con¬ 
verted  to  negative  samples.  This  process  is  then  repeated  for 
the  next  image/segmentation  pair,  with  updated  samples  inte¬ 
grated  into  the  classifier. 

To  classify  each  candidate  salient  region,  we  extract  a  fea¬ 
ture  vector  consisting  of  multiple  shape  and  intensity  prop¬ 
erties,  including 

-  The  total  area  of  the  region, 

-  The  minor  and  major  axis  length  of  the  ellipse  fit 
to  the  region, 

-  The  maximum  intensity  inside  the  region,  and 

-  The  mean  intensity  inside  the  region. 

These  properties  can  be  computed  quickly,  which  fits  well 
with  the  real-time  requirements  of  the  interactive  segmenta¬ 
tion  problem.  For  the  classifier,  we  use  a  SVM  with  a  RBF 
kernel  (y  =  0.01,  C  =  1.0),  which  can  be  retrained  while  a 
new  image  is  being  annotated.  Sample  salient  region  detec¬ 
tion  results  are  shown  in  Fig.  9,  with  true  positives  highlighted 
by  blue  bounding  boxes  and  false  positives  highlighted  by 
yellow  bounding  boxes.  We  can  see  that  false  positive  detec¬ 
tions  of  salient  regions  may  occur  when  there  are  strong  edge¬ 
like  noises  present  in  the  image.  The  performance  of  this 
classifier  on  the  initial  segmentation  of  the  Ti-21S  dataset, 
trained  on  128  previously  recorded  human  annotations,  is 
shown  in  Table  1 ,  where  FP,  FN,  TP,  and  TN  represent  false 
positives,  false  negatives,  true  positives  and  true  negatives, 
respectively. 


8  Implementation 

The  energy  minimization  components,  along  with  the  app¬ 
roach  from  Sect.  2,  are  implemented  in  Python  [36]  using 
the  NumPy/SciPy  [24],  scikit-leam  [34],  and  OpenCV  [7] 
libraries,  along  with  the  publicly  available  graph  cut  opti- 


mentation.  Based  on  the  interaction,  positive  and  negative  samples  can 
be  extracted  to  enhance  the  classifier  when  used  on  other  slices  in  the 
volume 


Fig.  9  Qualitative  results  of  salient  region  detection.  True  positive 
detections  of  salient  regions  are  surrounded  by  Mue  bounding  boxes. 
Re d  curves  are  the  detected  segment  boundaries  before  interactive  seg¬ 
mentation.  A  false  positive  detection  of  salient  regions  is  also  shown 
surrounded  by  a  yellow  bounding  box.  Image  intensity  inverted  for  clar¬ 
ity 

mization  (GCO)  [56]  library  based  on  [5,6,27].  The  inter¬ 
active  interface  is  built  as  a  web  application  using  the 
Django  [15]  web  framework  for  the  backend,  and  a  custom 
single-page  JavaScript  client  as  the  frontend.  In  the  follow¬ 
ing,  we  discuss  the  internal  architecture  that  enables  interac¬ 
tion  with  the  large  images  used  in  the  proposed  system,  along 
with  the  developed  interface. 

8.1  Architecture 

The  proposed  approach  is  implemented  using  a  client/server 
architecture  that  allows  different  frontend  interfaces  to  meet 
different  needs  for  interaction.  As  shown  in  Fig.  10,  our 
implementation  consists  of  a  large  disk  drive  as  a  datastore,  a 
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Table  1  Salient  region  detection  classifier  results  on  the  initial  segmen¬ 
tation  of  the  Ti-21S  dataset  after  training  on  128  previously  recorded 
human  annotations 


Slice 

FP 

FN 

TP 

TN 

Precision 

Accuracy 

1 

1 

1 

3 

17 

0.75 

0.91 

2 

1 

3 

7 

9 

0.88 

0.8 

3 

2 

2 

6 

22 

0.75 

0.88 

4 

1 

1 

11 

10 

0.92 

0.91 

5 

2 

6 

2 

30 

0.5 

0.8 

6 

1 

4 

1 

19 

0.5 

0.8 

7 

1 

3 

4 

5 

0.8 

0.69 

8 

1 

3 

7 

8 

0.88 

0.79 

9 

1 

4 

11 

15 

0.92 

0.84 

10 

1 

5 

10 

21 

0.91 

0.84 

Total 

12 

32 

62 

156 

0.84 

0.83 

Fig.  10  Overview  of  the  client/server  architecture  used  to  implement 
the  proposed  approach.  Large  datasets  are  persisted  on  disk  with  both 
the  underlying  image  (Ul )  and  segmentation  from  the  automatic  propa¬ 
gation  approach  (Sl )  saved  for  retrieval.  A  cache  allows  multiple  inter¬ 
actions  that  modify  the  segmentation  Sl  to  be  saved  in  memory,  where 
the  image  and  segmentation  can  be  quickly  retrieved  and  modified.  The 
client  may  explicitly  issue  a  “Save”  request  to  persist  changes  made  in 
the  cache  onto  disk 

cache  that  is  used  as  a  scratch  space,  and  a  client,  in  the  form 
of  a  web  application,  that  uses  a  REST  API  to  send  interac¬ 
tions  to  the  server.  Placing  all  the  data  on  a  hard  disk  allows 
large  materials’  images  to  be  stored.  However,  since  there 
may  be  more  than  one  interaction  required  for  a  materials 
image,  we  employ  an  in-memory  cache  between  the  client 
and  server  so  that  interactions  are  only  CPU-bound  and  are 
not  slowed  by  I/O  operations.  We  provide  an  explicit  “Save” 
operation  triggered  from  the  client  that  allows  changes  intro¬ 
duced  in  the  cache  to  be  made  permanent  on  disk.  In  addition, 
our  architecture  can  use  a  name- spaced  cache  to  safely  allow 
for  a  multiple-client  environment,  limited  only  by  the  cache 
size  and  CPU  usage  needed  to  run  the  energy  minimization 
for  each  interaction. 


8.2  Interface 

As  shown  in  Fig.  11,  the  client  consists  of  a  number  dis¬ 
tinct  areas  that  facilitate  interacting  with  a  selected  materials 
image,  obfuscating  any  client/server  communication  as  much 
as  possible. 

In  Fig.  1  la,  we  present  the  user  with  a  menu  of  tools  allow¬ 
ing  for  the  selection  of  interaction  type,  along  with  sliders  to 
control  the  parameters  of  the  addition  operation  in  Fig.  lib. 
We  allow  slice  selection  by  showing  thumbnails  of  each  slice 
in  Fig.  11c,  which  allows  the  user  to  choose  the  slice  to  oper¬ 
ate  on,  which  is  subsequently  displayed  in  the  designated 
display  area  shown  in  Fig.  lid  where  the  interactive  tools 
may  be  used.  We  also  record  log  output  in  Fig.  lie  listing 
actions  performed  by  the  client  so  that  all  interactive  sessions 
may  be  reproduced  as  needed.  Finally,  an  example  annotation 
is  shown  in  the  display  area  in  Fig.  Ilf. 

The  particular  tools  available  in  the  menu  (Fig.  11a)  are 
shown  in  Fig.  12.  Specifically,  we  include  a  selection  menu 
to  choose  the  particular  interactive  tool  in  Fig.  12a  (addi¬ 
tion,  removal,  addition  with  parameter  selection,  etc.).  We 
also  include  a  selection  menu  in  Fig.  12b  to  choose  how  the 
ij )  segmentation  is  displayed  on  the  raw  image,  and  whether 

the  salient  region  identification  is  shown  overlayed  on  top 
of  the  segmentation.  Commands  to  send  the  annotations  to 
the  server  are  shown  in  Fig.  12c,  where  different  commands 
perform  different  types/sequences  of  energy  minimization 
(only  local,  global  to  the  entire  image,  local  with  annotation 
repropagation,  etc.).  Finally,  we  include  a  menu  of  system 
tools  in  Fig.  12d  that  allow  the  user  to  erase  existing  annota¬ 
tions  (reset),  revert  the  work  from  the  cache  to  the  version  on 
disk  (reload),  and  save  work  from  the  cache  to  be  permanent 
on  disk  (save). 

An  annotator’s  workflow  consists  of:  (1)  loading  a  cho¬ 
sen  slice  using  Fig.  11c,  (2)  examining  this  slice  using  the 
display  area  in  Fig.  lid,  (3)  choosing  an  appropriate  anno¬ 
tation  tool  from  the  menu  in  Fig.  12a,  (4)  making  the  requi¬ 
site  annotation  in  the  display  area  as  shown  by  the  example 
in  Fig.  Ilf,  (5)  sending  annotations  to  the  server  with  the 
commands  in  Fig.  12c,  and  (6)  saving  the  work,  when  satis¬ 
factory,  with  the  menu  item  in  Fig.  12c.  Though  we  include 
keyboard  shortcuts  to  accelerate  this  workflow,  for  our  exper¬ 
iments  in  the  next  section,  we  explicitly  follow  this  workflow 
using  a  mouse,  and  record  all  mouse  clicks  made  to  do  the 
evaluation. 


9  Experiments 

9.1  /3-Ti  grains  in  Ti-2 IS 

To  evaluate  the  proposed  interactive  segmentation  method, 
we  use  it  to  segment  a  sequence  of  11  (indexed  from  0 
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Fig.  11  Client  interface  presented  to  the  user  for  interaction.  Consists 
of  a  nested  menus  that  present  interactive  tools  and  controls,  b  sliders  to 
control  parameters,  c  slice  selector  d  display  area  for  interaction,  e  log 


output  and  instruction  area,  and  f  addition  annotation  shown  displayed 
on  the  materials  image 


Fig.  12  Available  menu 
groups,  including  a  tool 
selection,  b  image  display 
modes,  c  commands  to  send 
annotations  to  server,  and  d 
system  tools  to  save  or  reload 
slices  from  the  server 
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Fig.  13  Evaluation  of  a  the 
amount  of  effort  (number  of 
clicks)  and  b  time  taken  for  a 
user  to  interactively  segment  the 
1 1  sample  slices.  Smaller  values 
are  better,  for  both  figures 


to  10)  microscopic  titanium  images  [40]  provided  by  Dave 
Rowenhorst,  NRL.  We  measure  the  effort  (i.e.,  number  of 
clicks)  used  to  segment  each  slice  in  the  dataset,  as  well  as 


the  overall  time  expended  by  the  user  to  segment  a  slice. 
The  previous  segmentation  propagation  approach  in  Sect.  2 
requires  a  complete  segmentation  on  one  slice  as  an  initializa- 
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(a)  F-measure 


(b)  Precision 


(c)  Recall 


Fig.  14  Performance  of  the  proposed  interactive  segmentation  method  compared  with  our  previous  automated  method  [58]  on  the  11  slices, 
measured  by  the  boundary  coincidence  with  the  ground  truth  segmentation  a  F-measure,  b  precision,  c  recall 


Fig.  15  Qualitative  results  where  each  subfigure  shows  the  initial  auto¬ 
matic  segmentation  Sv  ( left );  the  human  annotation  {middle)  with  the 
seed  pixels  in  green  and  dilation  pixels  in  yellow,  and  “X”s  indicat¬ 
ing  spurious  segments  to  be  removed;  and  the  updated  segmentation 

tion.  We  count  the  manual  segmentation  on  this  initial  slice 
into  the  effort  and  time  required.  We  present  the  proposed 
method  both  with  and  without  using  the  automatic  parame¬ 
ter  estimation  discussed  in  Sect.  6. 

For  comparison,  we  use  the  readily  available  “intelligent 
scissors”  interactive  segmentation  method  [32].  Using  the 
intelligent  scissors  tool,  we  independently  segment  all  11 
slices  from  the  same  dataset,  evaluating  both  effort  (number 


Sv  (right).  Note  that  f  and  g  illustrate  removal  annotation  and  the 
remaining  illustrate  addition  annotation.  Image  intensity  inverted  for 
clarity 


of  clicks)  and  time.  In  addition,  we  produce  a  hybrid  of  our 
previous  automatic  method  [58]  discussed  in  Sect.  2  and  the 
intelligent  scissors  method,  which  we  call  “intelligent  scis¬ 
sors  +  propagation”  in  Fig.  1 3 .  This  approach  uses  the  method 
from  Sect.  2  to  propagate  a  segmentation  from  an  initial  slice 
to  the  remaining  slices,  but  it  uses  the  intelligent  scissors 
tool  [32]  to  carry  out  the  interactive  component  instead  of 
the  interaction  proposed  in  this  paper. 
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Fig.  16  Synthetic  volume  generated  by  DREAM3D  [23]  to  evaluate 
the  proposed  interactive  segmentation 


The  results  of  this  comparative  experiment  are  shown  in 
Fig.  13.  Note  that,  in  propagated  methods  (“Proposed,”  “Pro¬ 
posed  +  Parameter  Estimation,”  and  “Intelligent  Scissors  + 
Propagation”),  the  first  slice  is  used  as  the  initial  slice  U,  so  it 
requires  significantly  more  effort  and  time  to  segment  com¬ 
pared  with  the  remaining  slices.  From  Fig.  1 3,  we  can  see  that 
the  method  proposed  in  this  paper  (“Proposed”)  allows  much 
more  rapid  segmentation  time  (<5  min  in  most  cases)  and 
with  much  less  effort  (<100  clicks  in  most  cases)  compared 
with  the  unpropagated  intelligent  scissors  method.  The  intel¬ 
ligent  scissors  method  (“Intelligent  Scissors”),  without  the 
benefit  of  propagation,  requires  significantly  more  time  and 
effort.  The  hybrid  method  (“Intelligent  Scissors  +  Propaga¬ 
tion”)  fares  better  than  the  unpropagated  intelligent  scissors 
method,  but  it  still  requires  greater  effort  than  the  proposed 
method.  Finally,  the  proposed  parameter  estimation  method 
(“Proposed  +  Parameter  Estimation”)  can  further  reduce  both 
the  time  and  effort  required  by  the  proposed  method. 

In  Fig.  14,  we  show  that  the  proposed  interactive  method 
is  able  to  increase  the  segmentation  accuracy  of  our  state-of- 
the-art  materials  image  segmentation  method  in  Sect.  2.  As 
in  our  previous  work  [58],  we  use  the  precision,  recall,  and 
F-measure,  which  is  the  harmonic  mean  of  the  precision  and 
recall  [31],  to  show  the  segment  boundary  coincidence  with 
the  manually  constructed  ground  truth  segmentation.  For 


both  the  proposed  and  previous  automatic  methods,  we  prop¬ 
agate  from  an  initial  slice  0  to  the  remaining  10  slices,  and 
the  proposed  interaction-enhanced  method  increases  perfor¬ 
mance  for  all  slices.  Finally,  qualitative  segmentation  results 
using  the  proposed  interactive  method  are  shown  in  Fig.  15 
where  we  show  the  automatic  segmentation  results  with  spu¬ 
rious  or  missing  segments,  the  human  annotation,  and  the 
updated  segmentation. 


9.2  DREAM3D  synthetic  volume 

We  further  evaluate  the  proposed  interactive  segmentation 
approach  by  generating  a  synthetic  grain  volume  using 
DREAM3D  [23].  As  shown  in  Fig.  16,  we  generate  a 
12  |xm  x  12  |xm  x  2  |xm  volume  where  each  slice  is 
0.06  |xm  x  0.06  \xm  x  0.1  |xm  in  size.  This  volume  is  sam¬ 
pled  as  a  200  x  200  x  20  image,  sliced  along  the  z-axis 
into  20  images  of  size  200  x  200  each.  We  use  these  20 
synthetic  slices,  along  with  a  noise  model  extracted  from 
the  above  Ti-21S  material,  as  additional  data  to  evaluate  the 
proposed  interactive  segmentation.  As  before,  we  evaluate 
interactive  methods  that  start  from  an  initial  segmentation, 
and  one  method  that  does  not.  For  methods  starting  from 
an  initial  segmentation,  we  use  the  results  of  propagating  the 
ground  truth  for  the  first  slice  to  all  the  remaining  slices  using 
the  approach  discussed  in  Sect.  2. 

Quantitative  results  of  this  evaluation  are  shown  in  Fig.  17. 
Included  in  this  evaluation  is  the  proposed  method  with  the 
parameter  estimation  discussed  in  Sect.  6  (“Proposed  +  Para¬ 
meter  Estimation”),  the  proposed  method  using  the  reprop¬ 
agation  method  discussed  in  Sect.  5  (“Proposed  +  Reprop¬ 
agation”),  along  with  the  intelligent  scissors  method  used 
to  completely  segment  every  slice  (“Intelligent  Scissors”) 
and  used  only  to  correct  the  propagated  results  (“Intelligent 
Scissors  +  Propagation”).  As  shown  in  Fig.  17a,  the  pro¬ 
posed  method  with  parameter  estimation  requires  less  effort 
compared  with  both  intelligent  scissors  methods.  With  the 


Fig.  17  Evaluation  on  the 
DREAM3D  synthetic  dataset 
showing  a  the  amount  of  effort 
(number  of  clicks)  and  b  time 
taken  for  a  user  to  interactively 
segment  the  20  sample  slices. 
Smaller  values  are  better  for 
both  figures 
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addition  of  the  repropagation  method,  interactions  are  only 
needed  on  three  slices,  since  the  corrections  are  then  reprop¬ 
agated  to  the  rest  of  the  volume,  resulting  in  significantly 
less  effort  than  all  other  methods.  Results  for  the  time  taken 
by  the  evaluated  methods  are  shown  in  Fig.  17b,  where  the 
intelligent  scissors  methods  all  require  more  time,  whereas 
the  proposed  method  with  repropagation  performs  better  than 
the  other  evaluated  methods. 


10  Conclusion  and  future  work 

We  have  presented  an  interactive  segmentation  method 
extended  from  our  automatic  segmentation  propagation 
approach.  By  allowing  the  user  to  interactively  handle  spuri¬ 
ous  and  missing  segments  when  propagating  from  one  slice 
to  another,  we  show  that  the  time  required  to  segment  a  mate¬ 
rials  image  volume,  as  well  as  the  overall  effort  (number  of 
clicks)  needed  for  interaction,  is  much  less  than  the  com¬ 
parison  “intelligent  scissors”  method  used  in  popular  image 
processing  tools.  By  updating  the  segmentation  within  a  local 
region  around  the  interactive  annotation,  we  are  able  to  obtain 
a  fast,  yet  robust  means  to  handle  segmentation  errors  when 
a  new  structure  appears  or  an  existing  structure  disappears 
from  the  2D  cross-section  of  a  particular  slice  in  the  volume. 
We  also  introduce  three  extensions  to  these  interactive  tools: 
an  annotation  repropagation  method  that  allows  interactions 
to  be  propagated  to  multiple  slices,  a  parameter  estimation 
technique  to  determine  the  seed  radius  when  adding  a  miss¬ 
ing  segment,  and  a  salient  region  detection  method  that  uses 
an  online  learning  approach  to  guide  an  annotator  using  the 
interactive  tools.  We  presented  the  client/server-based  web 
application  that  implements  these  tools  and  show  that  the 
proposed  approach,  extensions,  and  implementation  all  lead 
to  improved  performance. 

While  this  paper  is  focused  on  materials-science  image 
segmentation  for  underlying  grain  structures,  the  proposed 
interactive  method  may  be  extended  to  the  applications  in 
other  fields.  For  example,  many  3D  medical  images  [4,49, 
50],  such  as  MRI  and  CT  images,  are  also  taken  in  the  form  of 
a  sequence  of  serial-sectioned  2D  slices  with  good  structural 
continuity  between  adjacent  slices.  Furthermore,  for  some 
medical-imaging  applications  [16],  the  structures  to  be  seg¬ 
mented  are  of  large  number  with  specific  neighboring  rela¬ 
tions  and  they  share  high  similarity  to  the  grain  structures 
studied  in  this  paper.  In  the  future,  we  will  investigate  the 
extension  of  the  proposed  method  to  improve  the  accuracy 
of  medical  image  segmentation. 
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